User hand detecting device for detecting user&#39;s hand region and method thereof

ABSTRACT

Technology for a method of detecting a user hand by a user hand detecting device. The method according to an aspect of the present invention includes extracting a first mask image from a depth image in which the user hand is imaged; extracting a second mask image having a preset skin color value among regions corresponding to the first mask image in a color image in which the user hand is imaged; generating a skin color value histogram model in a color space different from a region of the color image corresponding to a color region of the second mask image; generating a skin color probability image of the different color space from the color image using the skin color value histogram model and an algorithm for detecting a skin color region; and combining the skin color probability image with the second mask image and detecting the user&#39;s hand region.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean PatentApplication No. 10-2014-0057774, filed on May 14, 2014, the disclosureof which is incorporated herein by reference in its entirety.

BACKGROUND

1. Field of the Invention

The present invention relates to technology for detecting a user's handregion in an image, and more specifically, to a user hand detectingdevice for detecting a user's hand region in smart glasses.

2. Discussion of Related Art

Recently, various companies including Google Inc (Google Glass) havereleased various optical see-through smart glasses. Unlike videosee-through head mounted displays (HMDs) in the related art, smartglasses can combine information observed by a user with an object of areal world and show the result.

As a method of interacting with content output to such smart glasses,methods in which a touch sensor, speech recognition, and gesturerecognition using camera vision are used are proposed. For example,Google Glass (Google Inc) uses a touch pad of an eyeglass frame andspeech recognition, and Space Glasses (Meta Company) uses a hand gesturerecognition method using a time of flight (ToF) camera. Among them, amethod of providing the most natural user interfaces/user experience(UI/UX) to a user is a method of performing an interaction using a handwith respect to content to be output.

Here, hand gesture recognition includes an accurate hand positiondetecting operation, a tracking and segmentation operation, and arecognition operation. In order to recognize a hand gesture accurately,the previous hand position detecting operation and tracking andsegmentation operation are very important.

Meanwhile, a camera-based hand gesture recognition method includes amethod in which color information obtained by a skin color model of acolor camera is used, a method in which 3D depth information of anobject obtained by a stereo camera, a ToF camera or the like is used,and a hybrid method in which two methods are combined.

Also, as a method in which skin color information is used in a colorimage, there are various methods such as a Gaussian mixture model.However, there is a problem in that a large amount of data is necessaryto generate skin color information, and detection is difficult based onunlearned data.

Also, a method of segmenting a hand position using a 3D depth image hasa problem in that it is difficult to segment a hand region precisely indepth information according to a change in an ambient environment (suchas a texture).

SUMMARY OF THE INVENTION

The present invention provides a user hand detecting device capable ofsegmenting a hand region robustly to a change in an ambient environmentin an image in which a user's hand is imaged, and a method thereof.

According to an aspect of the the present invention, there is provided auser hand detecting device for detecting a user's hand region, thedevice including an image acquiring unit configured to acquire a depthimage and a color image obtained by imaging a user hand; a first maskimage extracting unit configured to extract a first mask image from thedepth image (the first mask image includes an object within apredetermined distance from an imaging device that has generated thedepth image); a second mask image extracting unit configured to extracta second mask image having a preset skin color value among regionscorresponding to the first mask image in the color image; a skin colorprobability image generating unit configured to generate first andsecond skin color value histogram models in first and second colorspaces of a region of the color image corresponding to a color region ofthe second mask image, and generate a first skin color probability imageof the first color space and a second skin color probability image ofthe second color space from the color image using the first and secondskin color value histograms and an algorithm for detecting a skin colorregion (the first and second color spaces are different from the colorspace of the color image); and a hand region detecting unit configuredto combine at least two of the first skin color probability image, thesecond skin color probability image and the second mask image and detectthe user's hand region.

The color space of the color image may be an RGB color space, and thefirst and second color spaces may be a hue, intensity, saturation (HIS)color space and a YCbCr color space.

The second mask image extracting unit may extract a region correspondingto a preset Cb value and Cr value among regions corresponding to thefirst mask image in the color image as the second mask image.

The algorithm for detecting the skin color region may be a histogramback projection algorithm that uses a conditional probability anddetects a region having a high probability of being a skin color.

The hand region detecting unit may perform an OR operation of at leastone of the first and second skin color probability images and the secondmask image.

The hand region detecting unit may perform an AND operation of the firstskin color probability image and the second skin color probabilityimage, and perform an OR operation of the output image of the ANDoperation result and the second mask image.

According to another aspect of the present invention, there is provideda method of detecting a user hand by a user hand detecting device, themethod including: extracting a first mask image from a depth image inwhich the user hand is imaged (the first mask image includes an objectwithin a predetermined distance from an imaging device that hasgenerated the depth image); extracting a second mask image having apreset skin color value among regions corresponding to the first maskimage in a color image in which the user hand is imaged; generating askin color value histogram model in a color space different from aregion of the color image corresponding to a color region of the secondmask image; generating a skin color probability image of the differentcolor space from the color image using the skin color value histogrammodel and an algorithm for detecting a skin color region; and combiningthe skin color probability image with the second mask image anddetecting the user's hand region.

The different color space may be at least one of a hue, intensity,saturation (HIS) color space and a YCbCr color space.

The extracting of the second mask image may include extracting a regioncorresponding to a preset Cb value and Cr value among regionscorresponding to the first mask image in the color image as the secondmask image.

The algorithm may be a histogram back projection algorithm for detectinga region having a high probability of being a skin color using aconditional probability.

The detecting of the user's hand region may include detecting the user'shand region by performing an OR operation of the skin color probabilityimage and the second mask image.

According to still another aspect of the present invention, there isprovided a method of detecting a user hand by a user hand detectingdevice, the method including: extracting a first mask image from a depthimage in which the user hand is imaged (the first mask image includes anobject within a predetermined distance from an imaging device that hasgenerated the depth image); extracting a second mask image having apreset skin color value among regions corresponding to the first maskimage in a color image in which the user hand is imaged; generatingfirst and second skin color value histogram models in first and secondcolor spaces of a region of the color image corresponding to a colorregion of the second mask image (the first and second color spaces aredifferent from a color space of the color image); generating a firstskin color probability image of the first color space and a second skincolor probability image of the second color space from the color imageusing the first and second skin color value histograms and an algorithmfor detecting a skin color region; and combining at least two of thefirst skin color probability image, the second skin color probabilityimage and the second mask image and detecting the user's hand region.

The extracting of the second mask image may include extracting a regioncorresponding to a preset Cb value and Cr value among regionscorresponding to the first mask image in the color image as the secondmask image.

The first and second color spaces may be a hue, intensity, saturation(HIS) color space and a YCbCr color space.

The algorithm for detecting the skin color region may be a histogramback projection algorithm for detecting a region having a highprobability of being a skin color using a conditional probability.

The detecting of the user's hand region may include detecting the user'shand region by performing an OR operation of at least one of the firstand second skin color probability images and the second mask image.

The detecting of the user's hand region may include detecting the user'shand region by performing an AND operation of the first skin colorprobability image and the second skin color probability image andperforming an OR operation of the output image of the AND operationresult and the second mask image.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will become more apparent to those of ordinary skill in theart by describing in detail exemplary embodiments thereof with referenceto the accompanying drawings, in which:

FIG. 1 is a diagram illustrating a see-through smart glasses in which auser hand detecting device according to an embodiment of the presentinvention is implemented;

FIG. 2A and FIG. 2B are a diagram illustrating an exemplary image inwhich a user hand detected by an embodiment of the present inventioninteracts with content and the result is provided to a user;

FIG. 3 is a block diagram illustrating a user hand detecting device fordetecting a user's hand region according to an embodiment of the presentinvention;

FIG. 4 is a first exemplary diagram illustrating exemplary images fordescribing a method of detecting a user's hand region according to thepresent invention;

FIG. 5 is a second exemplary diagram illustrating exemplary images fordescribing a method of detecting a user's hand region according to thepresent invention;

FIG. 6 is a flowchart illustrating a method of detecting a user's handregion by a user hand detecting device according to an embodiment of thepresent invention; and

FIG. 7 is a flowchart illustrating a method of detecting a user's handregion by a user hand detecting device according to another embodimentof the present invention.

FIG. 8 is a block diagram illustrating a computer system for the presentinvention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Advantages and features of the present invention, and methods ofachieving the same will be clearly understood with reference to theaccompanying drawings and the following detailed embodiments. However,the present invention is not limited to the embodiments to be disclosed,but may be implemented in various different forms. The embodiments areprovided in order to fully explain the present invention and fullyexplain the scope of the present invention for those skilled in the art.The scope of the present invention is defined by the appended claims.Meanwhile, the terms used herein are provided to only describeembodiments of the present invention and not for purposes of limitation.Unless the context clearly indicates otherwise, the singular formsinclude the plural forms. It will be understood that the terms“comprises” or “comprising” when used herein, specify some statedcomponents, steps, operations and/or elements, but do not preclude thepresence or addition of one or more other components, steps, operationsand/or elements.

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings. First,when reference numerals are assigned to elements of each drawing, if thesame elements are illustrated in different drawings, the same referencenumerals are assigned to the same elements whenever possible. Also, indescriptions of the present invention, when detailed descriptions ofrelated known configurations or functions are deemed to unnecessarilyobscure the gist of the present invention, they will be omitted.

FIG. 1 is a diagram illustrating a see-through smart glasses 10 in whicha user hand detecting device according to an embodiment of the presentinvention is implemented.

As illustrated in FIG. 1, the see-through smart glasses 10 images a userhand through a camera 200 that is implemented in a predeterminedposition of an eyeglasses bridge. In this case, the user hand is imagedby a depth camera and a color camera.

The depth camera is a camera that is operated by a stereo method or acamera that is operated by a time of flight (TOF) method. A depth imagein which the user hand is imaged by the depth camera includes distanceinformation from the depth camera for each pixel.

The color camera is a camera using an image sensor such as a chargecoupled device (CCD) or a complementary metal oxide semiconductor(CMOS). A color image in which the user hand is imaged by the colorcamera includes color information for each pixel. In this case, thecolor image may be an image including color information expressed in ared, green, and blue (RGB) color space. Also, the color image may be animage including color information expressed in a YCbCr color space or ahue, intensity, and saturation (HIS) color space. In the presentinvention, the RGB color space color image will be exemplified.

Images (the depth image and the color image) in which the user hand isimaged are transmitted to a user hand detecting device 100 implementedin a predetermined position of an eyeglasses temple. In this case, theimages are transmitted to the user hand detecting device 100 throughwired or wireless communication.

The user hand detecting device 100 performs image processing on theimages transmitted from the camera 200 and detects the user hand in theimage. In this case, technology for detecting the user hand accuratelyin the image is necessary under various environments (such as light anda complex background).

Accordingly, the user hand detecting device 100 provides technology fordetecting the user hand using the depth image and the color image, anddetails thereof will be described below.

The smart glasses 10 may track the user hand's position detected by theuser hand detecting device 100, output the result through a lens 25, andprovide the result to the user. For example, as exemplified in FIG. 2Aand FIG. 2B, the user hand detected by the smart glasses 10 may interactwith content and the result may be provided to the user.

FIG. 3 is a block diagram illustrating a user hand detecting device fordetecting a user's hand region according to an embodiment of the presentinvention.

The user hand detecting device 100 according to the embodiment of thepresent invention extracts first and second mask images from the depthimage and the color image, generates a skin color probability imageexpressed in a color space different from the color image using theextracted second mask image, combines the generated skin colorprobability image with the second mask image, and detects the handregion.

For this purpose, as illustrated in FIG. 1, the user hand detectingdevice 100 according to the embodiment of the present invention includesan image acquiring unit 110, a mask image extracting unit 120, a skincolor probability image generating unit 130 and a hand region detectingunit 140. Hereinafter, the user hand detecting device 100 will bedescribed in detail with reference to FIGS. 4 and 5.

FIG. 4 is a first exemplary diagram illustrating exemplary images fordescribing a method of detecting a user's hand region according to thepresent invention. FIG. 5 is a second exemplary diagram illustratingexemplary images for describing a method of detecting a user's handregion according to the present invention.

The image acquiring unit 110 acquires the image in which the user handis imaged from the camera 200 of the smart glasses 10. Here, the imageacquiring unit 110 includes a depth image acquiring unit 111 and a colorimage acquiring unit 112. The depth image acquiring unit 111 acquires adepth image 41 in which the user hand is imaged from the depth camerathrough wired or wireless communication. Also, the color image acquiringunit 112 acquires a color image 42 in which the user hand is imaged fromthe color camera through wired or wireless communication.

The mask image extracting unit 120 uses the depth image 41 and the colorimage 42 acquired through the image acquiring unit 110 and extracts amask image representing the user's hand region approximately. Here, themask image extracting unit 120 includes a first mask image extractingunit 121 and a second mask image extracting unit 122.

The first mask image extracting unit 121 extracts a first mask image 43(coarse mask) from the depth image 41 acquired through the depth imageacquiring unit 111. In this case, the first mask image extracting unit121 extracts the first mask image 43 including an object within apredetermined distance (for example, 35 cm) from the depth camera in thedepth image 41.

This is because the hand position when the user wearing the smartglasses 10 outstretches his or her hand is highly likely to bepositioned within the predetermined distance. Here, the predetermineddistance can be changed by physical characteristics of the user.

The second mask image extracting unit 122 uses the color image 42acquired through the color image acquiring unit 112 and the first maskimage 43 extracted by the first mask image extracting unit 121 andextracts a second mask image 45 (fine mask). Here, the second mask imageextracting unit 122 extracts a region having a preset skin color valueamong regions corresponding to the first mask image 43 in the colorimage 42 as the second mask image 45.

For example, the second mask image extracting unit 122 may extract aregion having a preset skin color value among color regions of an image44 in which the color image 42 is mapped with the first mask image 43 asthe second mask image 45. Here, the preset skin color value may be apredetermined range value of a Cb value and a Cr value.

For this purpose, the second mask image extracting unit 122 may converta color space of an image using an image processing algorithm. Here, theimage processing algorithm is an algorithm for converting a color spaceof the color image 42 into a different color space. For example, theimage processing algorithm may be an algorithm for converting the colorimage 42 from an RGB color space into a YCbCr color space.

As an example, the second mask image extracting unit 122 may convertfirst the color space of the color image 42 acquired through the colorimage acquiring unit 112. That is, the second mask image extracting unit122 converts the color image 42 expressed in the RGB color space into animage expressed in the YCbCr color space and then maps the result withthe first mask image 43.

As another example, the second mask image extracting unit 122 may mapthe color image 42 with the first mask image 43 first, and then converta color space of the mapped image 44. In this case, the mask imageextracting unit 120 converts the mapped image 44 expressed in the RGBcolor space into an image expressed in the YCbCr color space.

When the color image acquired through the color image acquiring unit 112is an image expressed in the YCbCr color space rather than the RGB colorspace, the second mask image extracting unit 122 may omit a process ofconverting the color space.

The second mask image extracting unit 122 determines a Cb value and a Crvalue among a Y value, a Cb value, and a Cr value of the mapped image44, and extracts the second mask image 45. In this case, the second maskimage extracting unit 122 may extract a region in which a Cb value and aCr value are within a predetermined range in the mapped image 44 as thesecond mask image 45.

For example, the second mask image extracting unit 122 may extractregions satisfying at least one among three ranges of 85<Cb<127,137<Cr<177, and 183<Cb+0.6*Cr<219 in the mapped image 44 as the secondmask image 45. Preferably, the second mask image extracting unit 122 mayextract regions satisfying all of the above three ranges in the mappedimage 44 as the second mask image 45. Such predetermined ranges may bechanged and set by a worker in advance according to a skin colordifference for each ethnic group or the like.

The skin color probability image generating unit 130 generates a skincolor probability image in order to extract a hand region that is notincluded in the second mask image 45, that is, in order to increaseaccuracy of user hand detection.

The skin color probability image generating unit 130 generates first andsecond skin color value histogram models in first and second colorspaces with respect to a color region of the second mask image 45.

That is, the skin color probability image generating unit 130 generatesthe first and second skin color value histogram models in the first andsecond color spaces with respect to a color region in which the colorimage 42 is mapped with the second mask image 45. Here, the first andsecond color spaces are color spaces different from the color space ofthe color image 42. For example, when the color image 42 is assumed tohave a color signal according to the RGB color space, the first colorspace may be the HIS color space, and the second color space may be theYCbCr color space.

The skin color probability image generating unit 130 uses the first andsecond skin color value histogram models and an algorithm for detectinga skin color region, and generates first and second skin colorprobability images 52 and 53. Here, the first and second skin colorprobability images 52 and 53 refer to an extracted region (for example,a region having a probability of being a skin color that is apredetermined probability value or more) having a high probability ofbeing a skin color in the color image 42.

For example, the algorithm is a histogram back projection algorithm thatuses a conditional probability and detects a region having a highprobability of being a skin color in an image. The histogram backprojection algorithm is a method in which a color histogram is used toextract color features and a conditional probability value of an inputpixel is extracted. Specifically, the histogram back projectionalgorithm uses Equation 1, and estimates a posteriori probability valueof an input pixel in a predefined histogram.

$\begin{matrix}{{p\left( O \middle| C \right)} = {\frac{P(O)}{P(C)}{P\left( C \middle| O \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Here, O denotes an event in which a skin color of the user hand isexpressed, and C denotes an event in which there is a correspondingpixel. A conditional probability P(C|O) denotes a probability of a pixelbeing a color C when the pixel is extracted from a skin color of theuser's hand. On the other hand, a conditional probability P(O|C) denotesa probability of the pixel being extracted from the user hand.

Specifically, the skin color probability image generating unit 130generates a color value histogram of the color image 42 in the firstcolor space. Also, the skin color probability image generating unit 130uses the generated color value histogram, a first skin color valuehistogram model and a histogram back projection algorithm, detects aregion having a high probability of being a skin color in the colorimage 42, and generates a first skin color probability image 52.

Similarly, the skin color probability image generating unit 130generates a color value histogram of the color image 42 in the secondcolor space. Also, the skin color probability image generating unit 130uses the generated color value histogram, a second skin color valuehistogram model and a histogram back projection algorithm, detects aregion having a high probability of being a skin color in the colorimage 42, and generates a second skin color probability image 53.

The hand region detecting unit 140 combines the first and second skincolor probability images 52 and 53 generated in the skin colorprobability image generating unit 130 with the second mask image 45, anddetects the user's hand region.

As an example, the hand region detecting unit 140 combines the firstskin color probability image 52 with the second skin color probabilityimage 53 first. In this case, the hand region detecting unit 140acquires an image in which an AND operation of the first skin colorprobability image 52 and the second skin color probability image 53 isperformed. Here, the image on which the AND operation is performed is animage in which only a region common between the first skin colorprobability image 52 and the second skin color probability image 53 isextracted and noise is removed.

Also, the hand region detecting unit 140 combines the image in whichnoise is removed according to the AND operation result with the secondmask image 45, and acquires a hand region detection image 54. In thiscase, the hand region detecting unit 140 performs an OR operation of theimage in which noise is removed and the second mask image 45, andacquires the hand region detection image 54. Here, the hand regiondetection image 54 on which the OR operation is performed is an image inwhich all regions included in the image in which noise is removed orincluded in the second mask image 45 are extracted and detected as ahand region.

As another example, the hand region detecting unit 140 may combine (ORoperation) at least one of the first skin color probability image 52 andthe second skin color probability image 53 with the second mask image 45and detect a hand region.

In this manner, the user hand detecting device according to theembodiment of the present invention uses the depth image and the colorimage, extracts first and second mask images, generates a skin colorprobability image expressed in a color space different from the colorimage, combines the generated skin color probability image with a maskimage, and detects the user's hand region. Therefore, it is possible toincrease accuracy of technology for detecting the user hand in imagesunder various environments such as light and a complex background.

FIG. 6 is a flowchart illustrating a method of detecting a user hand bya user hand detecting device according to an embodiment of the presentinvention. Here, unless otherwise specifically described, the followingoperations are performed by the user hand detecting device 100.

In operation S610, a process of acquiring the depth image 41 and thecolor image 42 is performed. Here, the depth image and the color imageare acquired from the camera 200 of the smart glasses 10 (the depthcamera and the color camera). In this case, the depth image 41 and thecolor image 42 may be acquired from the camera 200 through wired orwireless communication.

In operation S620, a process of extracting the first mask image 43 fromthe depth image 41 is performed. Here, the first mask image 43 includesan object within a predetermined distance (for example, 35 cm) from thedepth camera in the depth image 41. This is because the hand positionwhen the user wearing the smart glasses 10 outstretches his or her handis highly likely to be positioned within the predetermined distance.Here, the predetermined distance can be changed by physicalcharacteristics of the user.

In operation S630, a process of extracting the second mask image 45using the first mask image 43 and the color image 42 is performed. Here,the second mask image 45 shows a region having a preset skin color valueamong regions corresponding to the first mask image 43 in the colorimage 42. For example, a region having a preset skin color value amongcolor regions of the image 44 in which the color image 42 is mapped withthe first mask image 43 may be extracted as the second mask image 45.Here, the preset skin color value may be a predetermined range value ofa Cb value and a Cr value.

For this purpose, an image processing algorithm may be used to convert acolor space of the color image 42. Here, the image processing algorithmis an algorithm for converting a color space when the color image 42 isobtained from the camera 200 into a different color space. For example,when the color image 42 is an image of the RGB color space, the imageprocessing algorithm converts the color image 42 from the RGB colorspace into the YCbCr color space.

The color image 42 in the RGB color space is converted into the image inthe YCbCr color space, and then a region having a preset skin colorvalue among regions corresponding to the first mask image 43 isextracted as the second mask image 45. In this case, a region in which aCb value and a Cr value of the mapped region among a Y value, a Cbvalue, and a Cr value are within a predetermined range may be extractedas the second mask image 45. Here, the predetermined range may bechanged and set by a worker in advance according to a skin colordifference for each ethnic group or the like. For example, a regionsatisfying ranges of 85<Cb<127, 137<Cr<177, and 183<Cb+0.6*Cr<219 in themapped region may be extracted as the second mask image 45.

In operation S640, a process of generating a skin color value histogrammodel in a different color space with respect to a color region 51 inthe second mask image 45 is performed. In this case, a skin color valuehistogram model in a color space different from the color space of thecolor image 42 acquired from the camera 200 is generated. For example,when the color image 42 is assumed to have a color signal according tothe RGB color space, the different color space may be the HIS colorspace or the YCbCr color space.

In operation S650, a process of generating a skin color probabilityimage expressed in the color space different from the color image 42using the skin color value histogram model is performed. Here, throughan algorithm for detecting a skin color region, a region (for example, aregion having a probability of being a skin color that is apredetermined value or more) having a high probability of being a skincolor in the color image 42 is extracted and a skin color probabilityimage is generated. In this case, the algorithm is a histogram backprojection algorithm for detecting a region having a high probability ofbeing a skin color in the image using Equation 1.

Also, a color value histogram of the color image 42 in a different colorspace (for example, the HIS or YCbCr color space) is generated. Then,the generated color value histogram, the skin color value histogrammodel generated in operation S640, and the histogram back projectionalgorithm are used, and a region having a high probability of being askin color in the color image 42 is generated as a skin colorprobability image.

In operation 5660, a process of combining the skin color probabilityimage with the second mask image 45 and detecting the hand region isperformed. For example, an OR operation of the skin color probabilityimage and the second mask image 45 is performed and the hand regiondetection image 54 is acquired. Here, through OR operation, all regionsincluded in the skin color probability image and the second mask image45 are extracted.

In this manner, the user hand detecting device according to theembodiment of the present invention uses the depth image and the colorimage, extracts first and second mask images, generates a skin colorprobability image expressed in a color space different from the colorimage, combines the generated skin color probability image with a maskimage, and detects the user's hand region. Therefore, it is possible toincrease accuracy of technology for detecting the user hand in imagesunder various environments such as light and a complex background.

FIG. 7 is a flowchart illustrating a method of detecting a user hand bya user hand detecting device according to another embodiment of thepresent invention. Here, unless otherwise specifically described, thefollowing operations are performed by the user hand detecting device100.

In operation S710, a process of acquiring the depth image 41 and thecolor image 42 is performed. Here, the depth image 41 and the colorimage 42 are acquired from the camera 200 of the smart glasses 10 (thedepth camera and the color camera) through wired or wirelesscommunication.

In operation S720, a process of extracting the first mask image 43 fromthe depth image 41 is performed. Here, the first mask image 43 includesan object within a predetermined distance (for example, 35 cm) from thedepth camera in the depth image 41. This is because the hand positionwhen the user wearing the smart glasses 10 outstretches his or her handis highly likely to be positioned within the predetermined distance.Here, the predetermined distance can be changed by physicalcharacteristics of the user.

In operation S730, a process of extracting the second mask image 45using the first mask image 43 and the color image 42 is performed. Here,the second mask image 45 shows a region having a preset skin color valueamong regions corresponding to the first mask image 43 in the colorimage 42. For example, a region having a preset skin color value amongcolor regions of the image 44 in which the color image 42 is mapped withthe first mask image 43 may be extracted as the second mask image 45.Here, the preset skin color value may be a predetermined range value ofa Cb value and a Cr value.

For this purpose, an image processing algorithm may be used to convert acolor space of the color image 42. Here, the image processing algorithmis an algorithm for converting a color space when the color image 42 isobtained from the camera 200 into a different color space. For example,when the color image 42 is an image of the RGB color space, the imageprocessing algorithm converts the color image 42 from the RGB colorspace into the YCbCr color space.

The color image 42 in the RGB color space is converted into the image inthe YCbCr color space, and then a region having a preset skin colorvalue among regions corresponding to the first mask image 43 isextracted as the second mask image 45. In this case, a region in which aCb value and a Cr value of the mapped region among a Y value, a Cbvalue, and a Cr value are within a predetermined range may be extractedas the second mask image 45. Here, the predetermined range may bechanged and set by a worker in advance according to a skin colordifference for each ethnic group or the like. For example, a regionsatisfying ranges of 85<Cb<127, 137<Cr<177, and 183<Cb+0.6*Cr<219 in themapped region may be extracted as the second mask image 45.

In operation S740, a process of generating a first skin color valuehistogram model in the first color space with respect to a color regionin the second mask image 45 is performed. In this case, a first skincolor value histogram model in the first color space different from thecolor space of the color image 42 acquired from the camera 200 isgenerated. For example, the first color space may be the HIS colorspace.

Similarly, in operation S750, a process of generating a second skincolor value histogram model in the second color space with respect to acolor region in the second mask image 45 is performed. In this case, asecond skin color value histogram model in the second color spacedifferent from the color space of the color image 42 acquired from thecamera 200 and the first color space is generated. For example, thesecond color space may be the YCbCr color space.

In operation S760, a process of generating the first skin colorprobability image 52 expressed in the first color space from the colorimage 42 using the first skin color value histogram model generated inoperation S740 is performed. Here, through an algorithm for detecting askin color region, a region (for example, a region having a probabilityof being a skin color that is a predetermined value or more) having ahigh probability of being a skin color in the color image 42 isextracted and the first skin color probability image 52 is generated. Inthis case, the algorithm is a histogram back projection algorithm fordetecting a region having a high probability of being a skin color inthe image. For example, the first skin color probability image 52 isobtained such that a color value histogram of the color image 42 in thefirst color space is generated, the generated color value histogram, thefirst skin color value histogram model and the histogram back projectionalgorithm are used, and a region having a high probability of being askin color in the color image 42 is extracted.

Similarly, in operation S770, a process of generating the second skincolor probability image 53 expressed in the second color space from thecolor image 42 using the second skin color value histogram modelgenerated in operation S750 is performed. Here, similar to operationS760, the histogram back projection algorithm is used. For example, thesecond skin color probability image 53 is obtained such that a colorvalue histogram of the color image 42 in the second color space isgenerated, the generated color value histogram, the second skin colorvalue histogram model and the histogram back projection algorithm areused, and a region having a high probability of being a skin color inthe color image 42 is extracted.

In operation 5780, a process of combining the first skin colorprobability image 52 generated in operation S760 with the second skincolor probability image 53 generated in operation S770 and removingnoise is performed. In this case, through an AND operation, only aregion common between the first skin color probability image 52 and thesecond skin color probability image 53 may be extracted and acquired asthe image in which noise is removed.

In operation 5790, the image in which noise is removed in operation 5780and the second mask image 45 are combined and the hand region isdetected. In this case, an OR operation of the image in which noise isremoved and the second mask image 45 is performed and the hand regiondetection image 54 may be acquired. Here, through the OR operation, allregions included in the image in which noise is removed or included inthe second mask image 45 are extracted.

In this manner, the user hand detecting device according to anotherembodiment of the present invention uses the depth image and the colorimage, extracts first and second mask images, generates first and secondskin color probability images expressed in a color space different fromthe color image, combines the generated first and second skin colorprobability images with a mask image, and detects the user's handregion. Therefore, it is possible to increase accuracy of technology fordetecting the user hand in images under various environments such aslight and a complex background.

The user hand detecting device according to the embodiment of thepresent invention uses a depth image and a color image to extract a maskimage, generates a skin color probability image expressed in a colorspace different from the color image, combines the generated skin colorprobability image with the mask image, and detects the user's handregion. Accordingly, it is possible to increase accuracy of technologyfor detecting the user hand in images under various environments such aslight and a complex background.

An embodiment of the present invention may be implemented in a computersystem, e.g., as a computer readable medium. As shown in FIG. 8, acomputer system 800 may include one or more of a processor 801, a memory803, a user input device 806, a user output device 807, and a storage808, each of which communicates through a bus 802. The computer system800 may also include a network interface 809 that is coupled to anetwork 810. The processor 801 may be a central processing unit (CPU) ora semiconductor device that executes processing instructions stored inthe memory 803 and/or the storage 808. The memory 803 and the storage808 may include various forms of volatile or non-volatile storage media.For example, the memory may include a read-only memory (ROM) 804 and arandom access memory (RAM) 805.

Accordingly, an embodiment of the invention may be implemented as acomputer implemented method or as a non-transitory computer readablemedium with computer executable instructions stored thereon. In anembodiment, when executed by the processor, the computer readableinstructions may perform a method according to at least one aspect ofthe invention.

The configuration of the present invention has been described above indetail through exemplary embodiments of the present invention, but itwill be understood by those skilled in the art that the presentinvention may be performed in other concrete forms without changing thetechnological scope and essential features. Therefore, theabove-described embodiments should be considered in a descriptive senseonly and not for purposes of limitation. The scope of the presentinvention is defined not by the detailed description but by the appendedclaims, and encompasses all modifications and alterations derived fromthe scope and equivalents of the appended claims.

REFERENCE NUMERALS

10: see-through smart glasses 100: user hand detecting device 110: imageacquiring unit 120: mask image extracting unit 130: skin colorprobability 140: hand region detecting unit image generating unit 200:camera 300: lens

What is claimed is:
 1. A user hand detecting device for detecting auser's hand region, comprising: an image acquiring unit configured toacquire a depth image and a color image obtained by imaging a user hand;a first mask image extracting unit configured to extract a first maskimage from the depth image—the first mask image includes an objectwithin a predetermined distance from an imaging device that hasgenerated the depth image; a second mask image extracting unitconfigured to extract a second mask image having a preset skin colorvalue among regions corresponding to the first mask image in the colorimage; a skin color probability image generating unit configured togenerate first and second skin color value histogram models in first andsecond color spaces of a region of the color image corresponding to acolor region of the second mask image, and generate a first skin colorprobability image of the first color space and a second skin colorprobability image of the second color space from the color image usingthe first and second skin color value histograms and an algorithm fordetecting a skin color region—the first and second color spaces aredifferent from the color space of the color image; and a hand regiondetecting unit configured to combine at least two of the first skincolor probability image, the second skin color probability image and thesecond mask image and detect the user's hand region.
 2. The deviceaccording to claim 1, wherein the color space of the color image is anRGB color space, and the first and second color spaces are a hue,intensity, saturation (HIS) color space and a YCbCr color space.
 3. Thedevice according to claim 1, wherein the second mask image extractingunit extracts a region corresponding to a preset Cb value and Cr valueamong regions corresponding to the first mask image in the color imageas the second mask image.
 4. The device according to claim 1, whereinthe algorithm for detecting the skin color region is a histogram backprojection algorithm that uses a conditional probability and detects aregion having a high probability of being a skin color.
 5. The deviceaccording to claim 1, wherein the hand region detecting unit performs anOR operation of at least one of the first and second skin colorprobability images and the second mask image.
 6. The device according toclaim 1, wherein the hand region detecting unit performs an ANDoperation of the first skin color probability image and the second skincolor probability image, and performs an OR operation of the outputimage of the AND operation result and the second mask image.
 7. A methodof detecting a user hand by a user hand detecting device, the methodcomprising: extracting a first mask image from a depth image in whichthe user hand is imaged—the first mask image includes an object within apredetermined distance from an imaging device that has generated thedepth image; extracting a second mask image having a preset skin colorvalue among regions corresponding to the first mask image in a colorimage in which the user hand is imaged; generating a skin color valuehistogram model in a color space different from a region of the colorimage corresponding to a color region of the second mask image;generating a skin color probability image of the different color spacefrom the color image using the skin color value histogram model and analgorithm for detecting a skin color region; and combining the skincolor probability image with the second mask image and detecting theuser's hand region.
 8. The method according to claim 7, wherein thedifferent color space is at least one of a hue, intensity, saturation(HIS) color space and a YCbCr color space.
 9. The method according toclaim 7, wherein the extracting of the second mask image includesextracting a region corresponding to a preset Cb value and Cr valueamong regions corresponding to the first mask image in the color imageas the second mask image.
 10. The method according to claim 7, whereinthe algorithm is a histogram back projection algorithm for detecting aregion having a high probability of being a skin color using aconditional probability.
 11. The method according to claim 7, whereinthe detecting of the user's hand region includes detecting the user'shand region by performing an OR operation of the skin color probabilityimage and the second mask image.
 12. A method of detecting a user handby a user hand detecting device, the method comprising: extracting afirst mask image from a depth image in which the user hand is imaged—thefirst mask image includes an object within a predetermined distance froman imaging device that has generated the depth image; extracting asecond mask image having a preset skin color value among regionscorresponding to the first mask image in a color image in which the userhand is imaged; generating first and second skin color value histogrammodels in first and second color spaces of a region of the color imagecorresponding to a color region of the second mask image—the first andsecond color spaces are different from a color space of the color image;generating a first skin color probability image of the first color spaceand a second skin color probability image of the second color space fromthe color image using the first and second skin color value histogramsand an algorithm for detecting a skin color region; and combining atleast two of the first skin color probability image, the second skincolor probability image and the second mask image and detecting theuser's hand region.
 13. The method according to claim 12, wherein theextracting of the second mask image includes extracting a regioncorresponding to a preset Cb value and Cr value among regionscorresponding to the first mask image in the color image as the secondmask image.
 14. The method according to claim 12, wherein the first andsecond color spaces are a hue, intensity, saturation (HIS) color spaceand a YCbCr color space.
 15. The method according to claim 12, whereinthe algorithm for detecting the skin color region is a histogram backprojection algorithm for detecting a region having a high probability ofbeing a skin color using a conditional probability.
 16. The methodaccording to claim 12, wherein the detecting of the user's hand regionincludes detecting the user's hand region by performing an OR operationof at least one of the first and second skin color probability imagesand the second mask image.
 17. The method according to claim 12, whereinthe detecting of the user's hand region includes detecting the user'shand region by performing an AND operation of the first skin colorprobability image and the second skin color probability image andperforming an OR operation of the output image of the AND operationresult and the second mask image.